Dynamic Factor Model Validation

Table 1: Input that aaron sent me

	Time	Pandemic_10	Pandemic_9	Pandemic_1	Pandemic_7
0	1/1/2020	0.000000	0.000000	0.000000	0.000000
1	2/1/2020	0.000000	0.000000	0.000000	0.000000
2	3/1/2020	0.005874	0.039735	0.004223	0.060936
3	4/1/2020	0.069314	0.019868	0.002413	0.028292
4	5/1/2020	0.137923	0.026490	0.002573	0.030468

Convert input into AnnData object

AnnData object with n_obs × n_vars = 36 × 4

Manually create DataProcessor object to confirm data/inputs match expectations

Columns that fail the ADF test (non-stationary)
['Pandemic_7']

Table 2: Values have not changed as they are already normalized between 0 and 1

	Pandemic_10	Pandemic_9	Pandemic_1	Pandemic_7
Time
1/1/2020	0.000000	0.000000	0.000000	0.000000
2/1/2020	0.000000	0.000000	0.000000	0.000000
3/1/2020	0.005874	0.039735	0.004223	0.060936
4/1/2020	0.069314	0.019868	0.002413	0.028292
5/1/2020	0.137923	0.026490	0.002573	0.030468

Display factor inputs to model

{'Pandemic_10': ('Pandemic',),
 'Pandemic_9': ('Pandemic',),
 'Pandemic_1': ('Pandemic',),
 'Pandemic_7': ('Pandemic',)}

Define and fit model

EM start iterations, llf=-148.46
EM iteration 10, llf=-142.49, convergence criterion=2.0537e-05
EM iteration 20, llf=-142.47, convergence criterion=6.7402e-06
EM iteration 30, llf=-142.46, convergence criterion=3.2861e-06
EM iteration 40, llf=-142.46, convergence criterion=1.9344e-06
EM iteration 50, llf=-142.46, convergence criterion=1.2716e-06
EM converged at iteration 57, llf=-142.46, convergence criterion=9.9125e-07 < tolerance=1e-06

/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/base/tsa_model.py:473: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

Save output

/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py:4250: FutureWarning:

Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value 'Pandemic_10    0.63
Pandemic_9     0.21
Pandemic_1     0.10
Pandemic_7     0.73
Name:    idiosyncratic: AR(1), dtype: object' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.

/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py:4250: FutureWarning:

Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value 'Pandemic_10    0.54
Pandemic_9     0.29
Pandemic_1     0.47
Pandemic_7     0.11
Name: var., dtype: object' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.

/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py:4316: FutureWarning:

Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '
Pandemic    0.77
Name:    error variance, dtype: object' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.

Table 3: Raw factors.csv output

	Factor_Pandemic	Factor_Pandemic_10	Factor_Pandemic_9	Factor_Pandemic_1	Factor_Pandemic_7
Time
1/1/2020	1.740749	0.000000	0.000000	0.000000	0.000000
2/1/2020	1.603046	0.000000	0.000000	0.000000	0.000000
3/1/2020	1.269691	0.005874	0.039735	0.004223	0.060936
4/1/2020	1.338951	0.069314	0.019868	0.002413	0.028292
5/1/2020	1.270113	0.137923	0.026490	0.002573	0.030468

Table 4: Normalized factors + inverted

	Factor_Pandemic	Factor_Pandemic_10	Factor_Pandemic_9	Factor_Pandemic_1	Factor_Pandemic_7	Invert_Factor_Pandemic
0	1.000000	0.000000	0.000000	0.000000	0.000000	0.000000
1	0.972549	0.000000	0.000000	0.000000	0.000000	0.027451
2	0.906095	0.005874	0.039735	0.004223	0.060936	0.093905
3	0.919902	0.069314	0.019868	0.002413	0.028292	0.080098
4	0.906179	0.137923	0.026490	0.002573	0.030468	0.093821

Merge factor result back into dataframe and plot

Figure 1: Result of input (already normalized between 0-1) and pandemic factor (global=0). Inverted factor fits data well.

Summary

Wow, John is such a cool dude